Papers with data synthesis pipeline
SLOT: Structuring the Output of Large Language Models (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Structured outputs are essential for large language models (LLMs) but often deviate from predefined schemas hampering reliable application development. |
| Approach: | They propose a model-agnostic approach that transforms unstructured LLM outputs into precise structured formats. |
| Outcome: | The proposed model-agnostic approach transforms unstructured LLM outputs into precise structured formats. |
DecIF: Improving Instruction-Following through Decomposition (2026.acl-long)
Copied to clipboard
| Challenge: | Existing approaches to obtain high-quality instruction-following data rely heavily on existing documents and existing methods. |
| Approach: | They propose a data synthesis framework, DecIF, which automatically generates accurate and diverse instruction-following data from scratch for supervised fine-tuning and reinforcement learning. |
| Outcome: | Extensive experiments show that the proposed framework can synthesize accurate instruction-following data for both SFT and RL paradigms compared to baselines. |
Synthesizing and Adapting Error Correction Data for Mobile Large Language Model Applications (2025.acl-industry)
Copied to clipboard
| Challenge: | Recent advances in large language models (LLMs) have achieved impressive performance on many language tasks. |
| Approach: | They synthesize a high-quality dataset of error correction pairs to evaluate and improve LLMs for mobile applications by reweighting the sample. |
| Outcome: | The proposed model improves on offline evaluation and live A/B testing, given the LLM performance on offline data and scores from a small privacy-preserving on-device language model. |
Mind the Query: A Benchmark Dataset towards Text2Cypher Task (2025.emnlp-industry)
Copied to clipboard
| Challenge: | Graph databases store data in nodes and relationships, enabling more natural modeling of complex, interconnected data. |
| Approach: | They present a high-quality dataset for the Text2Cypher task . it is enabling the translation of natural language (NL) questions into executable Cypher queries over graph databases. |
| Outcome: | The proposed dataset includes 27,529 NL queries and corresponding Cyphers spanning across 11 real-world graph datasets. |
Game Development as Human-LLM Interaction (2025.acl-long)
Copied to clipboard
| Challenge: | Currently, game development is a highly specialized task that relies on a complex game engine powered by complex programming languages, preventing many gaming enthusiasts from handling it. |
| Approach: | They propose a chat game engine powered by LLM that allows everyone to develop a custom game using natural language through Human-LLM interaction. |
| Outcome: | The proposed engine is designed to support the development of custom games using natural language through Human-LLM interaction. |
Unlocking Implicit Experience: Synthesizing Tool-Use Trajectories from Text (2026.acl-long)
Copied to clipboard
| Challenge: | Large language models (LLMs) can be used to effectively utilize tools in multi-turn interactions, but acquiring diverse and realistic multi-step tool-use data remains a challenge. |
| Approach: | They propose a text-based data synthesis pipeline that generates multi-turn tool-use trajectories from text corpora using relevance filtering, workflow tool extraction, trajectory grounding, and complexity refinement. |
| Outcome: | The proposed model achieves 14.9% improvement on the BFCL V3 Multi-turn benchmark while significantly reducing inference latency and costs. |
UI-E2I-Synth: Advancing GUI Grounding with Large-Scale Instruction Synthesis (2025.findings-acl)
Copied to clipboard
| Challenge: | Graphical User Interface (GUI) agents that utilize human-like vision perception capabilities are gaining a wider applicability compared to GUI metadata-based approaches. |
| Approach: | They propose a large-scale data synthesis pipeline for generating varying complex instruction datasets using GPT-4o instead of human annotators. |
| Outcome: | The proposed model achieves superior performance in GUI instruction grounding, demonstrating the advancements of proposed data synthesis pipeline. |
USB: A COMPREHENSIVE AND UNIFIED SAFETY EVALUATION BENCHMARK FOR MULTIMODAL LARGE LANGUAGE MODELS (2026.acl-long)
Copied to clipboard
Baolin Zheng, Guanlin Chen, Qingyang Teng, Hongqiong Zhong, Yingshui Tan, Zhendong Liu, Weixun Wang, Jiaheng Liu, Jian Yang, Huiyun Jing, Jincheng Wei, Wenbo Su, Xiaoyong Zhu, Bo Zheng, Kaifu Zhang
| Challenge: | Existing safety benchmarks fail to provide reliable assessments due to limited risk coverage, insufficient scale and the oversight of complex modality combinations. |
| Approach: | They propose a framework that covers 61 risk categories across four modality interactions to address this gap. |
| Outcome: | The proposed framework covers 61 risk categories across four distinct modality interactions. |
MR. Judge: Multimodal Reasoner as a Judge (2025.emnlp-main)
Copied to clipboard
| Challenge: | Effective reward modeling is especially valuable in reinforcement learning (RLHF) . |
| Approach: | They propose a paradigm for empowering general-purpose MLLMs judges with strong reasoning capabilities by using multiple-choice problem models instead of directly assigning scores. |
| Outcome: | The proposed model surpasses GPT-4o on VL-RewardBench and improves performance on MM-Vet by up to 7.7%. |
Psyche-R1: Towards Reliable Psychological LLMs through Unified Empathy, Expertise, and Reasoning (2026.acl-long)
Copied to clipboard
| Challenge: | Recent reasoning-augmented LLMs have demonstrated impressive capabilities across a wide range of domains owing to their exceptional text understanding capabilities. |
| Approach: | They propose a Chinese psychological LLM that integrates empathy, psychological expertise, and reasoning. |
| Outcome: | The proposed model produces over 75k high-quality psychological questions paired with detailed rationales, generated through and iterative prompt-rationale optimization procedure, along with 73k empathetic dialogues. |
Learning More from Less: Exploiting Counterfactuals for Data-Efficient Chart Understanding (2026.acl-long)
Copied to clipboard
Jianzhu Bao, Haozhen Zhang, Kuicai Dong, Bozhi Wu, Sarthak Ketanbhai Modi, Zi Pong Lim, Yon Shin Teo, Wenya Wang
| Challenge: | Chart understanding is a critical capability for vision-language models, serving as a cornerstone for automated data analysis, document understanding, and scientific research. |
| Approach: | They propose a chart-efficient training framework to enhance counterfactual sensitivity by code modification and a similarity-based data selection strategy. |
| Outcome: | The proposed framework achieves superior or comparable performance to strong chart-specific VLMs while using significantly less training data. |
Quantifying and Improving the Robustness of Retrieval-Augmented Language Models Against Spurious Features in Grounding Data (2026.acl-long)
Copied to clipboard
Shiping Yang, Jie Wu, Wenbiao Ding, Ning Wu, Shining Liang, Ming Gong, Hongzhi Li, Hengyuan Zhang, Angel X. Chang, Dongmei Zhang
| Challenge: | Existing studies on robustness to explicit noise (e.g., document semantics) but overlook implicit noise (spurious features). |
| Approach: | They propose a framework to quantify the robustness of RAGs against spurious features by integrating a data synthesis pipeline and a taxonomy. |
| Outcome: | The proposed framework quantifies the robustness of RALMs against spurious features. |
HSS-Synth: Humanities and Social Sciences Data Synthesis for LLMs (2026.findings-acl)
Copied to clipboard
Ru Peng, Tianyu Zhao, Xijun Gu, Zhiting Fan, Haokai Xu, Jinyang Zhang, Yawen Zeng, Yihong Zhuang, Kexin Yang, Junyang Lin, Dayiheng Liu, Junbo Zhao
| Challenge: | High-quality, diverse data are vital for large language models (LLMs) but remain scarce and costly. |
| Approach: | They define the first HSS domain system covering 14 mainstream fields and introduce HSS-Synth. |
| Outcome: | the proposed pipeline outperforms 14 leading baselines on 16 benchmarks. |